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1 .  SUMMARY 


1.1  Report  Period 

The  fifth  quarter  of  contract  activity  involved:  (1)  implementation 
of  changes  identified  during  the  initial  test  and  evaluation  experiments, 

(2)  specification  of  technical  improvements  and  extensions  for  the  Group 
Decision  Aid,  (3)  definition  of  a  cooperative  effort  with  NADC  to  apply 
the  aid  to  systems  planning  decisions,  and  (4)  specification  of  experimental 
procedures  for  evaluation  of  the  Group  Aid.  The  following  specific  tasks 
were  completed  during  the  report  period. 

(1)  A  series  of  changes  were  completed  following  the  test  and 
evaluation  experiments.  Included  were  (1)  consistency  checks 
on  probability  inputs,  (2)  correction  procedures  for  value 
changes  by  participants,  (3)  designs  for  decision  tree 
regeneration,  and  (4)  provision  for  audit  trail  time 
information. 

(2)  Possible  extensions  and  improvements  to  the  group  decision 
system  were  reviewed  for  the  period  from  the  present  to 
October  1,  1978.  A  set  of  high  priority  tasks  were  selected 
for  implementation,  dealing  with  scale  definition,  attribute 
weighting,  additional  sensitivity  and  conflict  analyses,  and 
new  procedural  options.  Tasks  planned  for  the  period 
following  October  1978  were  also  detailed. 

(3)  A  cooperative  effort  with  the  F-14  program  office  of  NADC  was 
established.  The  program  office  is  planning  to  use  the  Group 
Aid  for  problems  of  design  decisions,  procurement,  task 
scheduling,  and  other  areas  of  systems  planning. 


(4)  Plans  for  experimental  evaluation  of  the  group  decision  aid 
were  developed  in  cooperation  with  CACI.  Task  scenarios 
using  both  the  terrorist  crisis  and  NADC  systems  planning 
problems  were  included.  Measures  describing  group 
participation,  decision  development,  conflict  resolution, 
and  decision  quality  were  detailed. 

1.2  Next  Period 

The  contract  period  during  the  next  quarter  will  primarily 
concentrate  on  implementation  of  program  improvements  and  extensions. 
Informal  evaluations  of  these  improvements  will  then  be  made,  along  with 
finalization  of  plans  for  a  full-scale  system  evaluation.  The  specific 
items  of  work  for  the  next  period  include: 

(1)  Design,  implement,  and  test  system  improvements  and 
extensions. 

(2)  Evaluate  system  changes  using  informal  experimental  studies 
and  prepare  for  full-scale  experimental  studies. 

(3)  Continue  cooperative  effort  with  NADC  to  develop  system 
design  application. 


1.3  Program  Milestones 

The  milestone  chart  for  the  contract  program  is  shown  in  Figure  1-1, 
with  the  report  period  illustrated  as  the  checkered  portion. 


Program  Improvements 
Procedural  Improvements 
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TABLE  PROGRAM  MILESTONES 


Chapter  2  is  a  program  overview  containing  the  problem  statement, 
program  rationale,  and  program  objectives.  Chapter  3  describes  the  program 
improvements  and  extensions.  Chapter  4  is  an  analysis  of  the  planned 
cooperative  effort  with  NADC.  Chapter  5  describes  plans  for  experimental 
evaluation  of  the  Group  Decision  Aid. 


2.  PROGRAM  OVERVIEW 


2.1  Statement  of  Problem 


Constant  escalation  in  weapons  cost  and  effectiveness,  as  well  as 
the  increasing  complexity  of  international  relations,  makes  military 
decision  making  more  critical  today  than  ever  before.  In  today's  military 
environment,  most  upper-level  decisions  are  made  by  committees  and  staff 
groups.  Typically,  such  groups  contain  experts  from  several  speciality 
areas,  who  bring  to  the  decision  environment  disparate  sets  of  values. 
Decision  time  is  usually  limited,  the  decision  making  procedure  is 
relatively  unstructured,  and  intragroup  conflicts  arise  on  a  broad  variety 
of  issues.  Consequently  the  group  is  unable  to  consider  the  maximum  set 
of  alternatives,  conflicts  are  not  resolved  in  an  optimum  manner,  and  the 
resultant  decision  is  rarely  up  to  the  aggregate  potential  of  the  group 
membership. 

2.2  Rationale 

Decision  analysis  offers  a  promising  approach  to  solving  these 
problems.  The  analytical  procedure  of  building  a  decision  tree  formalizes 
the  decision  process,  and  permits  incorporation  of  individual  values 
(utilities)  into  the  selection  of  alternative  courses  of  action  (Hays, 
O'Connor,  and  Peterson,  1975).  However,  decision  analysis  as  it  is 
usually  practiced,  is  a  highly  personal  and  time-consuming  process. 
Decision  analysts  are  often  called  upon  to  assist  in  the  solution  of 
problems  ranging  over  a  large  variety  of  domains.  In  most  cases,  the 
decision  analysts  know  far  less  about  the  problem  domain  than  do  their 
clients.  Thus  their  contributions  are  confined  primarily  to  the  phases 
of  formalization  and  optimization.  While  optimization  is  usually  computer 
assisted,  the  formalization  phase  invariably  has  been  accomplished 
manually,  using  lengthy  interviews  of  persons  more  familiar  with  the 
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problem  area.  This  approach  is  generally  incompatible  with  the  conditions 
of  command  group  decision  making. 


Accordingly,  it  would  be  highly  worthwhile  to  automate  the 
formalization  phase,  using  an  interactive  computer  system  to  interrogate 
the  group  members  and  to  construct  a  decision  tree  based  on  their 
responses.  The  purpose  of  the  research  undertaken  here  is  to  develop  and 
evaluate  the  means  by  which  such  an  interactive  aid  could  be  used  to 
improve  group  decision  making. 

2.3  Objectives 

The  goal  of  the  research  program  addressed  in  this  prc  'ss 
report  is  to  develop  an  automated  decision  tree  elicitation  s  ^  using 
on-line  sensitivity  analysis  with  direct  real-time  group  feedbu^K  and 
evaluate  its  effectiveness  in  aiding  group  decision  making. 

The  specific  objectives  of  the  current  program  include  the 
following: 

(1)  Develop  computer  programs  for  efficient,  comprehensive, 
elicitation  of  decision  trees  from  a  decision  making  group. 

(2)  Develop  computer  programs  for  identifying  structural  and 
numerical  differences  among  the  contributions  of  individual 
group  members,  for  merging  these  contributions  and  for 
resolving  the  points  of  conflict. 

(3)  Develop  effective  means  for  displaying  to  the  group  the 
results  of  the  elicitation  procedures  and  conflict  analyses. 


(4)  Integrate  the  various  programs  and  techniques  into  a  complete 
aiding  system  which  can  be  readily  transferred  to  other  test 
environments. 

(5)  Experimentally  test  the  Group  Decision  Aid,  using  a  variety 
of  representative  military  decision  problems,  to  demonstrate 
its  advantages  under  realistic  conditions  of  use. 

(6)  On  the  basis  of  the  developmental  effort  and  the  experimental 
results,  establish  guidelines  and  recommendations  for  future 
military  applications  of  the  group  decision  aiding  methodology. 


3.  SYSTEM  IMPROVEMENTS 


3.1  General 

The  initial  system  tests,  described  in  our  recent  technical  report 
(Leal,  et  al,  1978),  were  highly  successful.  The  initial  development  goals 
of  the  system  were  realized,  in  that  experienced  groups  were  able  to  work 
effectively  with  the  aid  toward  the  solution  of  a  realistic  and 
representative  decision  problem.  At  the  same  time,  a  number  of  system 
areas  were  identified  in  which  existing  features  could  be  modified,  or  new 
features  added,  in  order  to  improve  overall  system  performance  and  power. 
Suggestions  for  candidate  improvements  came  (1)  from  our  review  of  original 
system  goals;  (2)  from  our  review  of  related  developments  in  computerized 
decision  aiding,  such  as  the  IBM  5100  aiding  packages  produced  for  ARPA/CTO 
by  DDI;  (3)  from  suggestions  made  by  knowledgeable  visitors  to  whom  the 
system  was  demonstrated,  such  as  Dr.  Clinton  Kelly,  of  DDI;  Dr.  Martin 
Tolcott  of  ONR;  Dr.  Ward  Edwards  of  USC,  and  others;  and  (4)  from 
suggestions  made  by  our  ARPA  monitors  and  other  ARPA  personnel. 

The  selection  and  ordering  of  the  improvements  were  derived  by 
evaluating  potential  tasks  with  respect  to  six  attributes:  (1)  effect  on 
decision  theoretical  fidelity,  (2)  effect  on  acceptability,  (3)  effect  on 
resources,  (4)  effect  on  group  processes,  (5)  effect  on  decision  quality, 
and  (6)  effect  on  decision  time.  Analysis  showed  that  a  number  of  the 
improvements  could  be  scheduled  for  development  during  the  FY  78  program 
period,  i.e.,  by  September  30,  1978.  Others,  requiring  a  greater 
expenditure  of  effort,  are  planned  for  the  following  period.  Both  sets 
of  improvements  are  described  in  the  following  sections. 
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Technical  Improvements 


The  following  extensions  to  the  Group  Decision  Aid  are  planned  to 
be  completed  by  October  1,  197o.  These  tasks  will  greatly  enhance  the 
decision-theoretic  fidelity  of  the  system  as  well  as  prepare  it  for  formal 
evaluations.  The  tasks  are  listed  in  order  from  easiest  to  hardest  to 
implement. 

Stored  Descriptions.  It  is  sometimes  difficult  to  capture  complex 
alternatives  or  outcome  situations  with  a  few  words  (currently  ten  letters 
for  tree  descriptions).  A  small  amount  of  time  should  be  used  by  the  group 
to  develop  detailed  definitions  and  descriptions  of  each  major  alternative, 
event,  and  attribute.  The  descriptions  may  be  displayed  at  any  time  for 
reference.  The  description  may  include  such  items  as  the  actor,  the  action, 
the  recipient,  the  situation,  and  the  time  frame.  An  example  of  an  action 
description  follows: 

Display  strength;  Deploy  carriers  immediately  to  Mandero  City 

and  await  response. 

Scale  Definitions.  It  is  critical  to  define  clearly  the  endpoints 
and  properties  of  the  utility  scales.  The  scales  of  both  the  attributes  and 
the  action/event  nodes  must  be  defined  consistently.  Each  attribute  will  be 
scaled  by  defining  the  worst  possible  level  to  be  zero  and  the  best  possible 
to  be  100.  Shortened  labels  should  be  assigned  at  each  of  these  endpoints 
and,  optionally,  at  the  midpoint  of  the  scale.  Quantitative  definitions, 
such  as  dollars  or  reliability  indices,  should  be  used  whenever  possible. 

An  example  scale  for  the  attribute  of  cost  follows: 

$100  million  $60  million  $20  million 

Loss _ Loss _ Loss 

0  50  100 
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The  overall  utilities  of  actions  or  events  must  be  defined  along  a 
single,  common  scale,  established  early  in  the  analysis.  Again,  the  zero 
point  will  correspond  to  the  worst  possible  state,  while  the  100  point  will 
denote  the  best  possible  state.  At  the  same  time,  consistency  will  be 
maintained  by  having  the  upper  and  lower  endpoints  correspond  to  all  zeros 
and  all  100s  on  the  individual  attributes,  respectively.  Key  points 
along  the  scale  (such  as  the  25,  50,  and  75  points)  will  be  described  by 
explicitly  defined  states. 

Adjustable  Attribute  List.  Where  conflicts  arise,  the  group  must 
assign  levels  to  the  established  attributes  of  the  particular  alternative 
in  conflict.  The  construction  of  the  attribute  list  is  the  first  task  the 
group  must  perform.  However,  it  is  very  difficult  to  foresee  the 
applicability  of  the  attributes  to  particular  future  alternatives  or 
outcomes.  An  adjustable  attribute  list  is  required  to  allow  new  attributes 
to  be  added  and  to  allow  irrelevant  ones  to  be  deleted. 

For  the  most  part,  the  attribute  set  should  be  invariant  with 
respect  to  the  alternatives.  This  is  because  the  attribute  membership 
derives  from  the  set  of  objectives  characterizing  the  problem,  and  not  from 
the  choices  present.  However,  development  of  the  decision  tree  may  bring 
to  light  additional  considerations  which  should  be  included  as  attributes. 
For  ease  of  comparability  across  actions,  such  changes  must  be  incorporated 
in  previously  evaluated  nodes. 

Attribute  Weighting.  It  is  unlikely  that  the  attributes  will  all 
be  equally  important  with  respect  to  the  particular  alternative  in  conflict. 
A  number  of  researchers  (Newman,  Seaver,  and  Edwards,  1976;  McClelland, 

1978)  have  shown  that  differential  weighting  (distinct  weights  for  each 
attribute)  is  important  in  most  real-world  situations.  The  evaluation 
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function  will  then  take  the  following  form: 


where  U.  is  the  overall  utility  of  outcome  j 

w 

w.  is  the  importance  weight  of  attribute  i 

A.,  is  the  level  of  attribute  i  occurring  with  outcome  j 

•  J 

The  weights  w^  define  the  policy  of  the  decision  maker  with  regard  to  the 
different  dimensions  of  outcome.  This  policy  should  be  invariant  with 
respect  to  the  possible  alternatives.  The  attribute  levels,  on  the  other 
hand,  characterize  the  choices  or  outcomes  along  the  dimensions,  costs, 
tactical  gains,  political  impact,  etc.  As  such,  these  levels  vary  with 

each  choice  and  must  be  estimated  for  each  point  in  conflict. 

The  importance  weights  will  be  elicited  from  each  participant 
after  definition  of  the  set  of  attributes.  The  elicitation  process  is 
patterned  after  that  used  by  DDI  (Selvidge,  1976)  and  by  SSRI  (Gardiner 
and  Edwards,  1975).  In  short,  comparisons  will  be  made  of  the  importance 
of  swings  across  the  range  of  each  attribute.  The  first  attribute  will 
arbitrarily  be  given  a  weight  of  ten  points.  Each  remaining  attribute 
will  be  compared  to  the  first  attribute  by  estimating  the  importance  of  a 
change  from  the  lowest  to  highest  level  compared  to  a  change  from  lowest 

to  highest  on  the  first  attribute.  For  example,  if  a  swing  across  the- 

range  of  the  first  attribute,  a  weight  of  20  will  be  assigned.  Once  all 
of  the  attributes  are  thus  weighted,  a  normalization  is  made:  each 
attribute  is  set  equal  to  its  raw  weight  divided  by  the  sum  of  raw  weights 
and  multiplied  by  100.  This  results  in  a  normalization  where  the  sum  of 
all  weights  equals  100  points. 


The  above  procedures  will  result  in  a  set  of  weights  for  each 
participant.  No  attempt  will  be  made  to  resolve  completely  the  individual 
differences  in  weighting,  as  these  differences  may  reflect  actual  conflicts 
in  policy  between  the  participants.  The  individual  weights  will  be  carried 
through  the  tree  development  and  conflict  resolution  processes.  In  this 
way,  conflicts  due  to  individual  policy  can  be  distinguished  from  conflicts 
due  to  differences  in  probability  or  attribute  level  estimation. 


It  was  noted  earlier  that  membership  of  the  attribute  set  should 
be  invariant  with  respect  to  the  part  of  the  tree  under  development. 
Similarly,  the  weighting  of  the  attributes  should  be  the  same  across  the 
tree.  However,  if  new  attributes  are  defined  or  old  ones  deleted,  the 
alternative  weighting  will  have  to  be  altered.  If  any  attribute  is 
deleted,  the  remaining  weights  can  be  renormalized  directly.  If  an 
attribute  is  added,  a  new  set  of  weights  must  be  elicited  and  renormalized 

Sensitivity  Analysis  Override.  It  may  be  counterproductive  to  be 
forced  to  expand  the  particular  node  "recommended"  by  the  sensitivity 
analysis  algorithm.  The  group  should  be  able  to  override  this 
recommendation  and  choose  any  node  to  expand.  This  feature  would  be  very 
helpful  during  demonstrations  and  filming  where  canned  sceanarios  are  used. 
Much  of  the  software  for  this  feature  has  already  been  implemented  in  the 
form  of  a  tree  traverser,  by  which  attention  may  be  directed  to  any  node. 
Also,  it  may  be  helpful  to  display  the  second  choice  recommended  by  the 
sensitivity  analysis  program.  This  information  will  be  provided  to  the 
intermediator. 

Interactive  Sensitivity  Analysis.  Sensitivity  analysis  has  many 
more  uses  than  simply  recommending  the  next  node  for  expansion.  For 
example,  probabilities  and  utilities  can  be  varied  across  their  possible 
ranges  to  determine  their  effect  on  the  original  decision.  Attribute 
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levels  and  weights  can  similarly  be  automatically  varied  to  see  if  they 
have  an  actual  impact  on  the  decision.  In  this  way,  attention  can  be 
shifted  from  considerations  of  minor  importance  to  factors  that  are 
critical  to  the  decision.  The  sensitivity  analysis  can  be  displayed 
graphically  or  by  showing  critical  points  of  the  parameters.  The  group 
members  should  be  able  to  request  such  sensitivity  modules  to  be 
exercised  by  the  intermediator.  Recent  work  has  been  done  in  this  area 
by  DDI  which  will  form  the  basis  for  development. 

Tree  Editor.  It  is  unrealistic  to  expect  the  group  to  be  able 
to  develop  a  decision  tree  without  error  the  first  time.  Alternatives 
will  be  forgotten;  events  will  be  found  too  improbable  to  be  included; 
formally  closed  branches  will  be  reopened,  etc.  A  tree  restructuring 
capability  will  permit  groups  to  alter  previously  built  portions  of  the 
tree  at  any  time.  This  capability  will  greatly  enhance  the  quality  of 
the  final  tree. 

New  Conflict  Resolution  Algorithm.  The  current  conflict 
identification  and  resolution  algorithm  is  somewhat  arbitrary  in  the 
sense  that  there  is  only  an  indirect  relationship  between  differences 
in  values  and  differences  in  preferred  actions.  For  example,  it  is 
possible  to  have  large  differences  in  opinion  as  to  the  value  of  an 
event,  but  agreement  on  the  initial  decision  (at  the  root  of  the  tree). 

A  new  algorithm  will  identify  conflicts  only  as  those  differences  of  . 
values  that  cause  a  difference  in  the  initial  decision.  The  sensitivity 
analysis  program  will  test  each  participant's  values,  one  at  a  time. 

This  new  conflict  resolution  algorithm  will  probably  be  employed  less 
frequently.  However,  when  it  is  used,  it  will  deal  with  "true"  conflicts. 

3.3  Deferred  Improvements 

The  following  extensions  are  planned  for  the  period  subsequent 
to  October  1,  1978.  These  tasks  incorporate  many  required  changes  and 
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additions  to  the  Group  Decision  in  order  to  produce  a  complete  and 
integrated  system. 

Prior  Information  Display.  It  is  vital  to  keep  the  group 
consistently  informed  about  their  progress.  This  information  includes 
the  current  region  of  the  decision  tree  under  discussion  as  well  as 
previous  utility  and  probability  assignments.  This  information  must  be 
displayed  during  each  elicitation  phase  and  not  just  at  the  end  of  each 
decision  cycle.  Methods  for  compact  and  comprehensive  information 
display  must  be  developed. 

Conflict  Resolution  Algorithm.  A  classification  of  conflict  with 
respect  to  source  (utilities,  probabilities,  or  attributes),  will  be 
helpful  toward  resolving  the  conflict.  This  is  planned  to  be  accomplished 
by  allowing  each  source  to  vary  while  holding  the  outer  sources  constant 
(setting  the  other  sources  to  their  average  across  the  participants). 

Then  if  a  conflict  is  present,  it  is  due  to  the  source  allowed  to  vary. 

Utility  and  Probability  Aggregation  Formulas.  An  analysis  of 
alternative  utility  and  probability  aggregation  formulas  will  provide  a 
basis  for  adopting  new  formulas  or  a  justification  for  retaining 
averaging  methods. 

N-Level  MAUM.  With  a  modest  amount  of  implementation  effort,  the 
conflict  resolution  module  containing  the  Multi -Attribute  Utility  Model 
(MAUM)  can  be  made  hierarchial  in  the  sense  that  individual  attributes 
can  be  decomposed  into  sub-attributes  when  in  conflict. 

Action/Event  Matrices.  In  cases  where  a  series  of  events  all 
apply  equally  to  a  series  of  possible  alternatives,  a  matrix 
representation  of  utilities  and  probabilities  may  be  more  efficient 


than  a  decision  tree  for  estimation.  DDI  has  developed  elicitation 
techniques  for  such  matrices  and  their  implementation  could  increase 
the  efficiency  and  quality  of  the  aid. 


Elicitation  Aids.  The  actual  assignment  of  utilities  and 
probabilities  can  be  enhanced  by  the  use  of  various  previously  developed 
techniques  that  require  the  decision  maker  to  analyze  the  assessment  in 
detail.  Such  techniques  utilize  internal  comparisons  and  validity  checks 
in  an  attempt  to  obtain  the  best  estimate  possible.  Although  the  aids 
usually  result  in  a  more  accurate  assessment,  they  take  a  considerable 
amount  of  time  to  execute.  Thoir  use  should  be  limited  to  particularly 
difficult  estimation  situations.  DDI  has  developed  some  elicitation  aids 
which  could  be  employed. 

Influence  Diagrams.  The  decision  tree  is  applicable  only  to 
decision  problems  that  can  be  expressed  in  terms  of  actions,  counter¬ 
actions,  and  probabilistic  events.  There  is  no  method  for  representing 
influences,  causes,  and  dependencies.  New  networks  called  "Influence 
Diagrams"  can  be  added  to  the  system  to  allow  a  wider  range  of  decision 
problems  to  be  accommodated. 

3.4  Bayesian  Probability  Aiding 

For  probability  estimates,  the  currently  configured  system  uses 
direct  estimation  along  a  0-100  scale.  This  method  is  simple  and 
effective  in  single-datum  situations  without  extreme  probabilities,  but 
may  lead  to  serious  inaccuracies  in  more  complex  circumstances.  A 
number  of  aiding  systems,  beginning  with  Edwards  (1962,  1964)  and  more 
recently  demonstrated  by  CACI  and  DDI,  ure  Bayes'  theorem  to  help  the 
decision  maker  optimally  revise  his  opinion  in  the  light  of  new 
information.  The  technique  allocates  data  evaluation  to  the  man  and 


data  aggregation  to  the  computer.  The  concept  requires  that  the  person 
estimate  conditional  probabilities  that  specific  data  will  be  observed 
when  certain  outcomes  occur,  and  then  transmit  these  to  the  computer, 
which  utilizes  Bayes'  theorem  to  aggregate  the  likelihood  estimates 
and  make  new  estimates.  By  following  this  procedure,  it  is  possible 
to  compensate  for  man's  inability  to  retain  and  combine  separate  data 
points  to  an  overall  conclusion.  Perceptronics  has,  in  fact,  developed 
such  a  system  for  hand-held  combat  input  of  multiple  likelihood  estimates 
(Ben-Bassat,  1978). 

Use  of  a  Bayesian  probability  aiding  program  would  provide  two 
functions:  (1)  it  would  result  in  more  accurate  estimation  of  complex 
probabilities,  and  (2)  availability  of  likelihood  estimate  input  would 
provide  a  valuable  check  on  the  simple  linear  scale  probability  inputs. 
For  these  reasons,  it  is  planned  to  incorporate  a  currently  operational 
Bayesian  module  (either  DDI  or  CACI)  with  the  Group  Decision  Aid. 
Feasibility  studies  of  each  system  will  be  conducted  during  the  current 
contract  period,  and  implementation  is  planned  to  take  place  during  the 
coming  program. 


4.  SYSTEMS  PLANNING  APPLICATION 


Several  meetings  have  taken  place  with  personnel  of  the  F-14 
program  office  of  NADC  toward  use  of  the  group  decision  aiding  system  for 
systems  planning.  The  program  office  is  highly  enthusiastic  about  applying 
the  aid  to  problems  of  design  decisions,  procurement,  task  scheduling,  and 
other  areas  of  system  planning.  Such  expansions  of  application  areas 
would  provide  ample  opportunities  for  extensive  evaluation  of  the  usefulness 
of  the  aid. 

A  typical  planning  problem  is  shown  in  Figure  3-1.  This  decision 
tree  is  representative  of  the  types  of  problems  that  would  be  encountered 
in  a  procurement  decision  situation.  The  initial  decision  node  shows  the 
problem  of  replacement  of  an  Inertial  Measurement  Unit  (IMU)  in  a  future 
aircraft  by  either  (1)  buying  an  outside  custom-made  unit,  (2)  buying  an 
existing  unit,  or  (3)  modifying  existing  equipment.  The  various  branches 
show  actions  and  events  as  they  may  be  developed  by  the  group. 

The  area  of  Systems  Planning  and  Design  Decision  has  some 
important  differences  from  that  of  Antierrorist  Actions.  The  clearest 
distinction  is  the  atmosphere  of  urgency  that  is  present  in  Anti -Terrorist 
decision  making  but  is  not  in  systems  planning.  Crisis  management  decisions 
are  made  in  hours  while  system  planning  decisions  usually  occupy  days  or 
even  weeks.  This  means  that  there  is  opportunity  to  develop  a  larger  and 
more  detailed  tree  with  many  more  possibilities  of  altering  previously 
constructed  portions.  The  "tree  editor"  then  becomes  a  more  important 
component  of  the  system.  Further,  there  is  time  to  receive  new  information 
from  the  outside  and  incorporate  it  into  judgmental  values  -  a  situation 
that  has  assumed  not  to  exist  in  previous  anti -terrorist  scenarios.  A 
program  that  calculates  the  cost  of  obtaining  new  information  and  its 
impact  on  the  decision  tree  would  be  very  beneficial.  Since  many  decisions 
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from  system  planning  deal  with  real  monetary  costs,  a  value  scale  based 
on  dollars  instead  of  utility  may  be  desirable.  This  would  provide  a 
basis  to  compare  the  value  of  various  situations  more  accurately. 

The  opportunity  to  store  specific  information  about  the  problem 
domain  is  enhanced  when  dealing  with  systems  planning.  Design  and 
procurement  decision  making  is  more  objective  than  anti -terrorist 
decision  making,  and  thus  would  provide  a  basis  for  a  more  exact  formulation 
of. previous  experience.  The  formation  of  a  domain-specific  data  base  for 
use  with  the  Group  Decision  Aid  would  be  not  only  more  straightforward, 
but  also  easier  to  access  and  use. 

The  effect  of  systems  planning  applications  on  experimental 
evaluation  is  enormous.  With  a  genuine  interest  on  the  part  of  NADC  to 
incorporate  the  Group  Decision  Aid  into  operational  design  procedures, 
the  opportunity  exists  to  compare  the  results  of  using  the  aid  with  (1) 
the  documented  outcomes  of  previous  decisions,  and  (2)  the  current 
decision-making  practices  at  NADC.  Such  an  environment  provides  a 
"ground  truth"  with  which  to  establish  objective  group  decision 
improvement  measures  -  an  opportunity  that  did  not  exist  iri  anti -terrorist 
decision  making. 


5.  SYSTEM  EVALUATION  PREPARATIONS 


5.1  Overview 


A  plan  for  comprehensive  evaluation  of  the  group  decision  aid  has 
been  developed  in  cooperation  with  CACI.  The  evaluation  will  be  directed 
toward  the  following  major  objectives: 

(1)  Determine  the  specific  contributions  to  problem  definition, 
conflict  resolution,  and  decision  quality  provided  by  the 
group  decision  aid. 

(2)  Aid  in  development  of  additional  system  capabilities. 

(3)  Determine  the  domain  of  application  of  the  system. 

(4)  Aid  in  production  of  system  specifications  and  guidelines 
for  operation. 

These  objectives  are  planned  to  be  accomplished  through  an  experimental 
program  based  on  comparisons  of  aided  and  unaided  group  performance  in 
several  different  task  scenarios.  The  following  sections  describe  the 
preparations  for  this  evaluation. 

5.2  Task  Scenarios 


Two  types  of  scenarios  are  planned  for  evaluation  of  the  group  aid 
The  first  is  the  currently  used  crisis  scenario  based  on  counter-terrorist 
actions.  The  CACI -developed  scenario  has  plausibility,  reasonable 
complexity,  and  exhibits  a  number  of  significant  judgmental  issues 
triggering  intra-group  conflict  (CACI,  1978).  It  has  also  been  the 


subject  of  extensive  analysis,  providing  baselines  for  analysis  of  group 
behavior.  As  such,  it  represents  an  ideal  vehicle  for  controlled 
experimentation.  The  second  scenario  is  the  NADC  systems  planning  design 
problems  described  earlier.  This  scenario  differes  from  the  terrorist 
situation  by  concentrating  on  monetary  consdierations  and  technological 
development  problems.  Also,  it  involves  deliberations  of  NADC  personnel 
on  actual  military  problems.  The  '•ADC  technical  design  scenario  has  not 
been  developed  to  the  level  of  the  terrorist  scenario,  but  should  provide 
a  useful  auxiliary  testbed  for  informal  studies. 

5.3  Performance  Measures 


A  variety  of  objective  and  subjective  measures  will  be  used  to 
evaluate  the  effectiveness  of  the  group  decision  aid.  The  measures  have 
been  refined  to  arrive  at  operational  procedures  and  practical  worksheets 
by  CACI  (1978).  These  measures  fall  into  four  main  categories:  group 
participation,  decision  development,  conflict  resolution,  and  decision 
quality.  Each  of  these  measures  will  be  discussed  in  detail. 

Group  Participation  Measures.  A  major  goal  of  the  group  aid  is 
to  increase  the  level  of  participation  in  the  decision  making  process. 

The  degree  of  participation  is  reflected  in  the  following  measures: 

(1)  Sturctural  inputs  -  the  number  of  alternatives,  events, 
and  attributes  submitted  by  each  member. 

(2)  Voting  deliberations  -  the  number  of  decision  makers 
participating  in  deliberations  about  probabilities  and 
attribute  levels. 


(3)  Temporal  participation  -  the  amount  of  time  each  decision 
maker  contributes  to  discussion. 


For  the  most  part,  these  measures  will  be  taken  from  recordings  of 
deliberations.  Audio  or  video  tape  media  will  provide  the  data. 

Decision  Development  Measures.  These  measures  evaluate  the  degree 
of  development  of  the  group  decision  output.  They  tend  to  be  observable 
and  easily  qualifiable. 

(1)  Tree  complexity  -  the  number  of  nodes,  the  average  number 
of  branches  emanating  from  each  node,  and  the  depth  of  tree 
in  number  of  levels. 

(2)  Information  used  -  the  completeness  of  use  of  the  information 
available  to  the  participants.  The  terrorist  scenario  has 
been  analyzed  by  experts  at  CACI,  developing  a  checklist  of 
critical  items  of  information  that  impact  the  occasion.  This 
checklist,  the  content  analysis  dictionary,  will  be  employed 
to  measure  the  completeness  of  information  use. 

(3)  Time  expenditures  -  the  time  spent  in  each  of  the  activities 
of  decision  structuring.  This  task  analysis  will  time  the 
activities  of  information  search,  data  incorporation, 
discussion  of  objectives,  synthesizing  alternatives, 
identifying  possible  events,  and  discussing  adversary 
capabilities.  Additional  non-group  activities  of  prompting 
by  the  intermediator  and  inputting  of  data  into  terminals 
will  also  be  logged.  Finally,  the  overall  time  for  completion 
of  decision  structuring  will  be  recorded. 

Conflict  Resolution.  The  degree  of  resolution  of  conflict  through 
use  of  the  group  aid  can  be  determiend  using  the  following  measures: 


(1)  Degree  of  change  after  discussion  of  utility,  probability, 
and  attribute  differences  among  members. 

(2)  Degree  of  change  in  action  evaluations  among  members  after 
discussion. 

Decision  Quality.  The  final  determinant  of  the  usefulness  of 
a  group  decision  aid  is  the  quality  of  the  decisions  provided.  While 
there  is  seldom  a  single  "correct"  solution  to  the  complex  judgmental 
problem  faced  by  the  groups,  a  number  subjective  and  quasi-objective 
indicators  of  decision  quality  can  be  incorporated. 

(1)  Structural  content  -  a  comparison,  node-by-node,  of  the 
tree  structure  and  parameter  estimates  with  a  "school 
solution".  The  school  solution  will  be  generated  by  expert 
personnel . 

(2)  Information  content  -  the  decision  tree  will  be  coded 
according  to  the  degree  of  consideration  of  a  set  of  key 
aspects  regarding  the  scenario.  CACI  has  prepared  such 

a  set  for  the  terrorist  scenario  in  the  form  of  a  "quality 
assessment  score  sheet".  An  overall  evaluation  then  results 
from  an  aggregation  of  the  coded  scores. 

(3)  Subjective  measures  -  subjective  ratings  will  be  recorded 
with  respect  to  decision  aid  performance  in  the  areas  of 
information  usage,  alternative  generation,  deliberation, 
evaluation,  choice  selection,  and  implementation. 

Taken  individually,  the  above  catalog  of  measures  describes  the  specific 
contributions  of  the  group  decision  aid.  The  measures  are  planned  to  be 
used  for  revision  and  refinement  of  the  aid  and  for  specification  of  the 


domain  of  application.  An  overall  evaluation  of  system  effectiveness  is 
more  difficult.  It  appears  that  such  an  evaluation  would  be  facilitated 
using  a  multi -attribute  scheme,  much  like  that  employed  by  the  aid  itself. 
The  four  areas  of  measurement  and  their  associated  indices  form  a 
hierarchical  evaluation  function.  Each  type  of  measurement  thus  represents 
a  separate  attribute.  Weighting  and  aggregation  of  these  attributes 
according  to  the  objectives  of  the  task  situation  then  provide  an  overall 
evaluation. 
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